Deep Neural Networks for Multi-Label Text Classification: Application to Coding Electronic Medical Records

Coding Electronic Medical Records (EMRs) with diagnosis and procedure codes is an essential task for billing, secondary data analyses, and monitoring health trends. Both speed and accuracy of coding are critical. While coding errors could lead to more patient-side financial burden and misinterpretation of a patient’s well-being, timely coding is also needed to avoid backlogs and additional costs for the healthcare facility. Therefore, it is necessary to develop automated diagnosis and procedure code recommendation methods that can be used by professional medical coders. The main difficulty with developing automated EMR coding methods is the nature of the label space. The standardized vocabularies used for medical coding contain over 10 thousand codes. The label space is large, and the label distribution is extremely unbalanced - most codes occur very infrequently, with a few codes occurring several orders of magnitude more than others. A few codes never occur in training dataset at all. In this work, we present three methods to handle the large unbalanced label space. First, we study how to augment EMR training data with biomedical data (research articles indexed on PubMed) to improve the performance of standard neural networks for text classification. PubMed indexes more than 23 million citations. Many of the indexed articles contain relevant information about diagnosis and procedure codes. Therefore, we present a novel method of incorporating this unstructured data in PubMed using transfer learning. Second, we combine ideas from metric learning with recent advances in neural networks to form a novel neural architecture that better handles infrequent codes. And third, we present new methods to predict codes that have never appeared in the training dataset. Overall, our contributions constitute advances in neural multi-label text classification with potential consequences for improving EMR coding

Ordinal Convolutional Neural Networks for Predicting RDoC Positive Valence Psychiatric Symptom Severity Scores

Background—The CEGS N-GRID 2016 Shared Task in Clinical Natural Language Processing (NLP) provided a set of 1000 neuropsychiatric notes to participants as part of a competition to predict psychiatric symptom severity scores. This paper summarizes our methods, results, and experiences based on our participation in the second track of the shared task. Objective—Classical methods of text classification usually fall into one of three problem types: binary, multi-class, and multi-label classification. In this effort, we study ordinal regression problems with text data where misclassifications are penalized differently based on how far apart the ground truth and model predictions are on the ordinal scale. Specifically, we present our entries (methods and results) in the N-GRID shared task in predicting research domain criteria (RDoC) positive valence ordinal symptom severity scores (absent, mild, moderate, and severe) from psychiatric notes. Methods—We propose a novel convolutional neural network (CNN) model designed to handle ordinal regression tasks on psychiatric notes. Broadly speaking, our model combines an ordinal loss function, a CNN, and conventional feature engineering (wide features) into a single model which is learned end-to-end. Given interpretability is an important concern with nonlinear models, we apply a recent approach called locally interpretable model-agnostic explanation (LIME) to identify important words that lead to instance specific predictions. Results—Our best model entered into the shared task placed third among 24 teams and scored a macro mean absolute error (MMAE) based normalized score (100 · (1 − M M AE)) of 83.86. Since the competition, we improved our score (using basic ensembling) to 85.55, comparable with the winning shared task entry. Applying LIME to model predictions, we demonstrate the feasibility of instance specific prediction interpretation by identifying words that led to a particular decision

A Marker-based Neural Network System for Extracting Social Determinants of Health

Objective. The impact of social determinants of health (SDoH) on patients' healthcare quality and the disparity is well-known. Many SDoH items are not coded in structured forms in electronic health records. These items are often captured in free-text clinical notes, but there are limited methods for automatically extracting them. We explore a multi-stage pipeline involving named entity recognition (NER), relation classification (RC), and text classification methods to extract SDoH information from clinical notes automatically. Materials and Methods. The study uses the N2C2 Shared Task data, which was collected from two sources of clinical notes: MIMIC-III and University of Washington Harborview Medical Centers. It contains 4480 social history sections with full annotation for twelve SDoHs. In order to handle the issue of overlapping entities, we developed a novel marker-based NER model. We used it in a multi-stage pipeline to extract SDoH information from clinical notes. Results. Our marker-based system outperformed the state-of-the-art span-based models at handling overlapping entities based on the overall Micro-F1 score performance. It also achieved state-of-the-art performance compared to the shared task methods. Conclusion. The major finding of this study is that the multi-stage pipeline effectively extracts SDoH information from clinical notes. This approach can potentially improve the understanding and tracking of SDoHs in clinical settings. However, error propagation may be an issue, and further research is needed to improve the extraction of entities with complex semantic meanings and low-resource entities using external knowledge

Linguistic Elements of Engaging Customer Service Discourse on Social Media

Customers are rapidly turning to social media for customer support. While brand agents on these platforms are motivated and well-intentioned to help and engage with customers, their efforts are often ignored if their initial response to the customer does not match a specific tone, style, or topic the customer is aiming to receive. The length of a conversation can reflect the effort and quality of the initial response made by a brand toward collaborating and helping consumers, even when the overall sentiment of the conversation might not be very positive. Thus, through this study, we aim to bridge this critical gap in the existing literature by analyzing language's content and stylistic aspects such as expressed empathy, psycho-linguistic features, dialogue tags, and metrics for quantifying personalization of the utterances that can influence the engagement of an interaction. This paper demonstrates that we can predict engagement using initial customer and brand posts.Comment: Accepted to NLP+CSS at EMNLP 202

Adversarial Discriminative Domain Adaptation for Extracting Protein-Protein Interactions from Text

Relation extraction is the process of extracting structured information from unstructured text. Recently, neural networks (NNs) have produced state-of-art results in extracting protein-protein interactions (PPIs) from text. While multiple corpora have been created to extract PPIs from text, most methods have shown poor cross-corpora generalization. In other words, models trained on one dataset perform poorly on other datasets for the same task. In the case of PPI, the F1 has been shown to vary by as much as 30% between different datasets. In this work, we utilize adversarial discriminative domain adaptation (ADDA) to improve the generalization between the source and target corpora. Specifically, we introduce a method of unsupervised domain adaptation, where we assume we have no labeled data in the target dataset

A Comprehensive Study of Gender Bias in Chemical Named Entity Recognition Models

Objective. Chemical named entity recognition (NER) models have the potential to impact a wide range of downstream tasks, from identifying adverse drug reactions to general pharmacoepidemiology. However, it is unknown whether these models work the same for everyone. Performance disparities can potentially cause harm rather than the intended good. Hence, in this paper, we measure gender-related performance disparities of chemical NER systems. Materials and Methods. We develop a framework to measure gender bias in chemical NER models using synthetic data and a newly annotated dataset of over 92,405 words with self-identified gender information from Reddit. We applied and evaluated state-of-the-art biomedical NER models. Results. Our findings indicate that chemical NER models are biased. The results of the bias tests on the synthetic dataset and the real-world data multiple fairness issues. For example, for synthetic data, we find that female-related names are generally classified as chemicals, particularly in datasets containing many brand names rather than standard ones. For both datasets, we find consistent fairness issues resulting in substantial performance disparities between female- and male-related data. Discussion. Our study highlights the issue of biases in chemical NER models. For example, we find that many systems cannot detect contraceptives (e.g., birth control). Conclusion. Chemical NER models are biased and can be harmful to female-related groups. Therefore, practitioners should carefully consider the potential biases of these models and take steps to mitigate them

Defunding Law Enforcement: Fire Departments\u27 Perspective on Implementing the National Fire Protection Association 3000 Standard When Preparing for an Active Shooter Mass Casualty Incident

This study applied the policy window theory through punctuated equilibrium and resource dependency theories to analyze the perception of nationwide fire service leaders and the impact that defunding law enforcement can have on the fire service in managing an active shooter mass casualty incident (ASMCI). As police reform remains the center of discussion throughout the nation, many community leaders have explored ways to re-appropriate police funding. This comes at a time when the paradigm of law enforcement and Fire/Emergency Medical Services (EMS) interdependency has become the standard response to ASMCIs as defined by the National Fire Protection Association 3000 Standard for an active shooter event. Using John Creswell\u27s (2018) approach to mixed-methods design, a nationwide survey was sent to 1352 fire departments with open and closed-ended questions to measure their perception of ASMCI joint training and response impact. Survey data was collected, and through parametric testing, results were converged with qualitative data. This research explored the perception of the fire service in training and response to an ASMCI through the reliance on law enforcement and whether the fire service could evolve its response practices to address any delay in ASMCI response as outlined in NFPA 3000. The results reveal that fire officials regard training as a preparation tool to address the threat of an ASMCI and recognize that the community would expect the fire service to explore new models to evolve their role if required. This research area is emergent to policy discourse as the movement to defund law enforcement or funding reform can affect fire/EMS in managing an ASMCI emergency

BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?

Language models have seen significant growth in the size of their corpus, leading to notable performance improvements. Yet, there has been limited progress in developing models that handle smaller, more human-like datasets. As part of the BabyLM shared task, this study explores the impact of reinforcement learning from human feedback (RLHF) on language models pretrained from scratch with a limited training corpus. Comparing two GPT-2 variants, the larger model performs better in storytelling tasks after RLHF fine-tuning. These findings suggest that RLHF techniques may be more advantageous for larger models due to their higher learning and adaptation capacity, though more experiments are needed to confirm this finding. These insights highlight the potential benefits of RLHF fine-tuning for language models within limited data, enhancing their ability to maintain narrative focus and coherence while adhering better to initial instructions in storytelling tasks. The code for this work is publicly at https://github.com/Zephyr1022/BabyStories-UTSA.Comment: Accepted to BabyLM workshop at CoNL

Análisis de las tácticas del marketing relacional a través de la teoría del Comportamiento Planificado durante la pandemia del 2020. Estudio de caso de la empresa Rappi

Durante los últimos años, el sector de delivery por aplicativo ha tenido un crecimiento exponencial en nuestro país. Con el pasar del tiempo, han podido ingresar nuevos competidores haciendo que la oferta de este servicio sea cada vez más competitiva. Sin embargo, con la llegada de la pandemia, en el año 2020, se generaron cambios estructurales en el mercado, haciendo que el servicio se viera inicialmente paralizado por unos meses. Ante ello, el Estado comenzó a desarrollar políticas públicas orientadas a impulsar una reactivación de la economía duramente castigada por el cierre, permitiendo así la operación de este servicio nuevamente. Este iba a afrontar un cambio en el comportamiento del consumidor local, así como nuevas medidas protocolares de bioseguridad. La presente investigación se propuso describir la intención de compra por parte de los clientes de Rappi durante la pandemia a través de la Teoría del Comportamiento Planificado. Para ello, como medio de información, se utilizaron las tácticas del marketing relacional, con el fin de describir si las acciones de Rappi tuvieron efectos sobre la intención de compra. De la misma manera, se tuvo en cuenta el contexto de inmovilización ciudadana obligatoria al momento de analizar el comportamiento del consumidor hacia las aplicaciones y servicios dedelivery. Para ello, se desarrollaron entrevistas semiestructuradas, principalmente, a clientes de la aplicación Rappi, así como a expertos en los temas de marketing relacional y sobre la Teoría del Comportamiento Planificado, finalmente, a trabajadores administrativos de la empresa. En segundo lugar, se realizaron un análisis de la información recolectada para determinar la influencia de las tácticas sobre la intención de compra. En función a la información recolectada, se determinó que Rappi influyó a través de cuatro de las cinco tácticas el comportamiento del consumidor durante la pandemia. Cabe resaltar que en algunas dimensiones tuvo mayor influencia dado sus distintas aproximaciones con las tácticas realizadas